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ABSTRACT 


Data  representation,  which  is  often  overlooked  in  many  image  processing  and  analysis 
applications,  is  as  critical  as  the  algorithms  applied  to  that  data.  When  data  is  represented 
properly,  simple  algorithms  can  be  much  more  powerful  than  sophisticated  and  often  com¬ 
plex  algorithms  applied  to  an  improper  representation.  In  an  image,  the  useful  information 
is  generally  mixed  in  with  irrelevant  information  or  noise  and  often  it  is  difficult  for  the 
computer  to  separate  the  useful  information  from  the  large  volume  of  irrelevant  data  with¬ 
out  destroying  much  of  the  useful  data.  To  provide  a  solid  foundation  for  a  good  solution 
to  this  problem,  “multiresolution  decomposition  and  synthesis”  approaches  have  been 
developed  which  decompose  raw  image  data  into  a  set  of  partial  information  channels, 
where  each  channel  represents  a  certain  modality  (or  aspect)  of  the  raw  image.  The  chan¬ 
nels  can  then  be  processed  individually  or  cooperatively  with  a  wide  variety  of  results  pos¬ 
sible.  After  processing  of  the  channels  is  completed,  they  can  be  selectively  synthesized  to 
recower  the  original  image,  an  improved  version,  or  an  image  with  certain  features  high¬ 
lighted.  In  this  way,  the  irrelevant  information  can  be  more  effectively  rejected  or  ignored 
while  the  useful  data  can  be  used  or  modified  in  whichever  manner  is  desired.  For  this 
project,  I  have  studied  the  usefulness  of  the  nonlinear  image  pyramid,  compared  to  the  lin¬ 
ear  type  of  pyramid  and  raw  data  format,  in  the  areas  of  image  compression,  image  trans¬ 
forms,  and  image  enhancement  via  noise  removal. 
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ABSTRACT 


Data  representation,  which  is  often  overlooked  in  many  image  processing  and  analysis 
applications,  is  as  critical  as  the  algorithms  applied  to  that  data.  When  data  is  represented 
properly,  simple  algorithms  can  be  much  more  powerful  than  sophisticated  and  often  com¬ 
plex  algorithms  applied  to  an  improper  representation.  In  an  image,  the  useful  information 
is  generally  mixed  in  with  irrelevant  information  or  noise  and  often  it  is  difficult  for  the 
computer  to  separate  the  useful  information  from  the  large  volume  of  irrelevant  data  with¬ 
out  destroying  much  of  the  useful  data.  To  provide  a  solid  foundation  for  a  good  solution 
to  this  problem,  “multiresolution  decomposition  and  synthesis”  approaches  have  been 
developed  which  decompose  raw  image  data  into  a  set  of  partial  information  channels, 
where  each  channel  represents  a  certain  modality  (or  aspect)  of  the  raw  image.  The  chan¬ 
nels  can  then  be  processed  individually  or  cooperatively  with  a  wide  variety  of  results  pos¬ 
sible.  After  processing  of  the  channels  is  completed,  they  can  be  selectively  synthesized  to 
recover  the  original  image,  an  improved  version,  or  an  image  with  certain  features  high¬ 
lighted.  In  this  way,  the  irrelevant  information  can  be  more  effectively  rejected  or  ignored 
while  the  useful  data  can  be  used  or  modified  in  whichever  manner  is  desired.  For  this 
project,  I  have  studied  the  usefulness  of  the  nonlinear  image  pyramid,  compared  to  the  lin¬ 
ear  type  of  pyramid  and  raw  data  format,  in  the  areas  of  image  compression,  image  trans¬ 
forms,  and  image  enhancement  via  noise  removal. 


1.0  Introduction 

Multiresolution  decomposition  techniques  are  commonly  used  in  variety  of  signal  pro¬ 
cessing  applications  [Salembier  26],  including  but  not  limited  to  image  compression,  pro¬ 
gressive  transmission,  enhancement,  restoration,  analysis,  et  c.[Ranganath  426],  The 
classical  approach  to  image  decomposition  has  been  the  repeated  applications  of  a  linear, 
usually  Gaussian,  type  of  lowpass  filter  to  obtain  a  frequency  sensitive  decomposition. 
However,  in  many  cases  this  may  not  be  the  best  choice  as  some  applications  are  better 
suited  for  a  size  or  shape  dependant  decomposition.  Thus,  instead  of  a  decomposition 
resulting  in  a  division  of  frequency  space,  it  may  be  desirable  to  have  a  decomposition  of 
the  pattern  spectrum. 

When  our  research  began,  our  aim  was  to  investigate  the  applicability  of  morphological 
decompositions  to  the  area  of  image  compression.  It  is  well  known  that  the  linear  type  of 
pyramid  can  yield  compact  image  code;  could  the  nonlinear  decomposition  lead  to  compa¬ 
rable  or  even  better  general  compression  results? 

Image  compression  can  also  be  achieved  through  the  use  of  image  transforms,  such  as  the 
Discrete  Cosine  transform,  K-L  transform.  Single  Value  Decomposition,  et  c.  Transform 
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coding,  also  known  as  block  quantization,  involves  a  unitary  transformation  of  each  block 
so  that  a  large  fraction  of  the  total  energy  is  packed  into  relatively  few  transform  coeffi¬ 
cients.  The  image  can  then  be  reconstructed  from  these  few  coefficients  with  a  minimal 
amount  of  error.  The  next  step  was  to  examine  the  possibility  that  the  image  pyramids 
could  be  more  effectively  transformed  than  the  transformation  of  the  original  image  with 
respect  to  image  compression. 

Finally,  the  linear  image  decompositions  can  also  be  used  for  image  enhancement  and  res¬ 
toration.  Restoration  of  noisy  images  and  undesirable  artifact  removal  are  just  two  of  the 
possible  areas  that  the  pyramid  structure  can  be  effectively  used.  Also,  because  of  the  mul¬ 
tiscale  nature  of  the  pyramids,  some  types  of  filtering  can  be  done  much  more  efficiently 
on  the  pyramid  planes  than  on  the  raw  image  data  [Ranganath  437].  In  this  area,  the  possi¬ 
bility  that  morphological  pyramids  could  outperform  the  linear  pyramids  and  equivalent 
types  of  operations  on  raw  image  data  was  investigated. 


2.0  General  Pyramid  Concept 

The  pyramid  image  decomposition  technique  is  an  iterative  procedure  designed  to  isolate 
image  features  at  different  scales  and  to  support  more  efficient  scaled  neighborhood  oper¬ 
ations  through  reduced  image  representation  [Lee  4].  In  this  section,  an  overview  of  how 
the  pyramids  are  constructed  and  then  how  the  image  can  be  synthesized  from  the  data  _ 
will  be  given.  Possible  types  of  pyramid  processing  are  discussed  in  great  detail  in  a  later 
section.  For  the  general  explanation  of  the  pyramid  structure,  terminology  for  linear  pyra¬ 
mids  will  be  used  as  it  is  more  widely  understood. 

2.1  General  pyramid  Structure 

First,  a  lowpass  decomposition  of  the  original  image  is  accomplished.  The  zeroth  level  of 
the  lowpass  pyramid  will  be  the  original  image,  I.  The  image  is  then  (linearly  or  morpho¬ 
logically)  filtered  and  down-sampled  to  obtain  the  next  level.  This  is  applied  repetitively 
to  obtain  the  entire  lowpass  pyramid. 


=  / 


(Li  =  D(F(Li_  j))),i=>l 


*D(.)’  represents  the  down-sizing  operation  which  is  simply  a  decimation  by  two  in  each 
dimension,  so  for  a  two  dimensional  image,  the  down-sized  image  has  1/4  the  pixels  of  its 
parent.  *F(.)’  is  the  filtering  operation,  usually  convolution  for  the  linear  case  and  some 
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type  of  morphological  operation  (Erosion,  Dilation,  Closing,  Opening,  et.c.)  for  the  non¬ 
linear  application. 

The  bandpass  pyramid  is  then  generated  using  the  lowpass  pyramid  by  simply  subtracting 
each  lowpass  decomposition  by  the  next  lower  level  in  the  lowpass  pyramid.  However,  the 
lower  level  differs  in  size  by  a  factor  of  2  in  each  dimension.  So,  the  image  must  be  up- 
sized,  U(.),  which  is  a  point  replication  operation,  and  then  interpolated,  P(.).  The  interpo¬ 
lator  can  be  of  whatever  type  the  user  desires.  Often  for  linear  pyramids,  a  Gaussian  type 
of  lowpass  filter  is  used,  while  the  same  or  a  morphological  interpolator  can  be  used  for 
the  nonlinear  pyramids.  Thus  the  bandpass  decomposition,  Bi,  can  be  specified  in  terms  of 
the  lowpass  decompositions.  These  bandpass  images  are  often  referred  to  as  prediction 
error  or  residuals  because  they  are  not  true  bandpass  images  due  to  the  errors  introduced 
by  the  non-ideal  filters  and  interpolator.  Figure  1  graphically  shows  the  construction  pro- 


B,  =  L(-  P(U  (Li+  ,) ) 


cedures  for  both  the  lowpass  and  bandpass  pyramids. 


FIGURE  1.  Construction  procedures  for  the  lowpass  and  bandpass  decompositions 
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FIGURE  2.  2-D  Spatial  Frequency  /  Pattern  Spectra  Plot  of  Bandpass  Pyramid 


Figure  2  shows,  graphically,  how  the  bandpass  pyramid  will  ideally  look  as  the  bandpass 
decomposition  of  an  image  yields  different  bands  of  either  spatial  frequency  or  morpho¬ 
logical  pattern  spectrum  [Kelm  3].  It  must  be  noted  that  due  to  errors  introduced  by  inter¬ 
polation  and  less  than  ideal  filters  the  actual  frequency/pattem  spectra  plot  will  not  be  this 
clean.  One  large  advantage  the  pyramid  structure  has  over  some  other  types  of  representa¬ 
tions,  such  as  a  Fourier  transform,  is  that  the  bandpass  images  retain  their  spatial  loc<uity. 
Shapes  of  interest  or  even  single  objects  can  be  enhanced  without  modifying  the  rest  of  the 
image. 

One  important  property  of  this  type  of  image  representation  is  that  the  image  is  repre¬ 
sented  completely.  Full  reconstruction  is  possible  without  using  all  of  both  pyramids.  As 
shown  in  Figure  1 ,  an  order  ‘k’  pyramid  will  have  k  distinct  levels,  Lq  — >  ,  in  the  low- 

pass  pyramid  and  k  levels,  Bq  —  >  B^.j,  in  the  bandpass  pyramid.  However,  only  the 
smallest  lowpass  image,  L*  ,  and  the  bandpass  pyramid  are  needed  for  full  image  recon¬ 
struction.  It  will  later  be  proven  that  at  most  4/3  of  the  pixels  in  the  original  image  are 
required  to  store  the  entire  bandpass  pyramid  and  . 
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FIGURE  3.  Course  to  fine  synthesis  process 


Figure  3  shows  the  image  synthesis  process  which  recovers  either  the  original  or  improved 
version  from  the  bandpass  pyramid,  B0  -->  Bk_2,  and  the  smallest  lowpass  image  Lk.j 
which  are  shown  as  darkened  images.  The  reconstruction  process  starts  with  up-sizing, 
U(.),  Lk_j  and  then  interpolating  the  larger  version.  The  next  step  is  to  add  in  the  smallest 
of  the  bandpass  pyramids,  Bk-2  to  obtain  Lk-2.  These  steps  are  then  repeated  all  the  way 
up  the  pyramid  until  reconstruction  is  completed. 


L-  =  (Bj  +  P  (U  (Li+  j) ) )  forall i  ,i=l  —  >N-2,and 


/  =  L0  =  B0  +  P((/(L,)) 


This  method  of  reconstruction,  course  to  fine  synthesis,  nicely  lends  itself  to  a  controlled 
synthesis  process.  For  small  2-D  images  this  is  not  a  real  concern,  but  when  3-D  imaging 
is  attempted  the  controlled  synthesis  could  likely  save  the  user  considerable  time  and 
effort  when  examining  say  3-D  medical  data.  Controlled  synthesis  works  in  the  following 
manner:  first  at  the  coarse  resolution,  volumes  of  interest  are  identified;  as  the  synthesis 
progresses  only  those  regions  identified  as  interesting  are  fully  synthesized;  the  end  result 
being  that  the  user  now  has  a  3-D  image  with  the  sections  deemed  interesting  at  the  finest 
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resolution  and  the  areas  not  of  concern  are  ‘roughed  out.’  This  would  be  ideally  suited  for 
the  progressive  transmission  of  3-D  image  data. 

2.2  Burt’s  Laplacian  Pyramid 

The  Laplacian  pyramid  proposed  by  Burt  and  Adelson  in  1983  uses  the  identical  construc¬ 
tion  process  discussed  above.  Their  goal  was  to  use  the  pyramid  structure  as  a  method  for 
image  compression  and  they  did  get  some  decent  results,  over  5  to  1  on  USC.girl  with 
only  0.88%  mean  squared  error  and  nearly  11  to  1  on  the  ‘Walter’  image  with  only  0.43% 
mean  squared  error.  To  achieve  this  result  they  further  reduced  the  entropy  in  by  quantiz¬ 
ing  the  pixel  values  in  each  level  of  the  pyramid  images  used  in  reconstruction.  I  used  their 
‘best’  pyramid  as  a  benchmark  for  my  nonlinear  pyramids  so  a  short  overview  of  their 
work  is  required. 

As  previously  stated,  the  general  image  pyramid  construction  procedures  are  identical  to 
the  process  discussed  earlier.  In  fact  the  only  difference  between  their  ‘best’  pyramid  and 
all  of  my  nonlinear  pyramids  is  the  filter,  F(.),  applied  to  the  image  to  generate  the  lowpass 
pyramid.  All  the  other  operations  are  identical. 

The  filter  they  chose  to  use  is  a  5  x  5  weighted  average  which  attempts  to  minimize  the 
error  after  a  level  is  down-sized,  up-sized,  and  then  interpolated.  One  dimensionally,  Fig¬ 
ure  4  shows  a  graphic  representation  at  the  pixel  level  of  the  Gaussian  pyramid  generation 
process. 


L2 


LI 


I  =  Lo 


FIGURE  4. 1-D  graphic  representation  of  Gaussian  pyramid  generation  process 


The  parameters  ‘3’,  ‘b’,  and  ‘C’  are  the  one  dimensional  weighting  values  used  in  the  5  x 
5  convolution.  The  mask  ‘w(m,n)’  is  designed  to  be  separable  and  symmetric  with  respect 
to  the  center  of  the  mask,  m=0  and  n=0.  Their  second  stipulation  was  that  every  pixel  at  a 
given  level  must  contribute  the  same  total  weight  (1/4)  to  the  next  level’s  nodes,  equal 


August  22,  1991 


7 


contribution.  This  is  satisfied  when  a  +  2c  =  2b.  It  should  also  be  noted  that  the  mask  is 
normalized  to  1.  These  constraints  are  satisfied  when:  (w(0)  =  a),  (w  (1)  =  w  (-1)  =•  1/4), 
(w  (2)  =  w  (-2)  =  1/4  -  3/2).  So  the  only  option  in  their  design  is  what  value  for  a  will  be 
used.  When  3=0.4,  this  turns  out  to  be,  after  repetitive  application,  a  reasonable  approxi¬ 
mation  to  a  Gaussian  function  and  as  the  value  for  3  decreases  the  function  becomes  more 
flat.  The  best  results  they  obtained  was  when  the  value  for  a  was  chosen  to  be  0.6.  The 
function  is  actually  trimodel  in  nature  when  a  >  0.5. 

2.3  Nonlinear  Pyramids 

Now  that  the  general  pyramid’s  and  Burt’s  Laplacian  pyramid’s  construction  procedures 
have  been  discussed,  we  turn  to  the  nonlinear  pyramids.  The  construction  procedures  for 
the  nonlinear  pyramids  are  no  different  than  the  general  construction  techniques.  The  only 
difference  again  being  the  filter  used  to  generate  the  lowpass  pyramid. 

2.3.1  Basic  Morphological  Operators 

For  the  benefit  of  any  who  actually  read  this  report  and  may  not  be  familiar  with  the  basic 
morphological  operations,  I  will  give  a  very  brief  overview  of  the  main  operators  used. 
There  are  two  simple  operators  which  define  most  morphological  operators  or  filters,  ero¬ 
sion  and  dilation.  Let  B  represent  the  structuring  element  and  I  be  the  image.  The  gray¬ 
scale  dilation  and  erosion  of  image  I  by  structuring  element  B  can  then  be  defined  as: 

Dilation  =  I®B  =  MAXx  y  e  B  (/f  +  x  j  +  y  +  Bx  y)  (forall  (ij  e  /) ) 


Erosion  =  IQB  =  MIN  R 


(/ 


i  +  x,j  +  y 


~BX  y)  (forall  (ij  e  /)) 


These  definitions  are  assuming  the  structuring  element  is  symmetric  with  respect  to  the 
center  of  B  and  B’s  center  is  defined  to  be  x  =  0  and  y  =  0.  For  simplicity,  the  erosion  by 
structuring  element  B  will  be  wriaen  as  Eg(.)  and  dilation  as  Dg(.).  By  combining  these 
two  operators,  other  morphological  operators  can  be  defined.  The  opening  operator  is  sim¬ 
ply  an  erosion  followed  by  a  dilation  and  closing  is  a  dilation  followed  by  an  erosion. 


Opening  =  0R  (I)  =  Dg(Efi(/)) 
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Closing  =  Cq  (/)  =  Eb(Db(I )) 


2.3.2  Morphological  Filters 

Any  one  or  combination  of  the  above  morphological  operations  can  be  used  as  the  filter  to 
generate  the  decomposition  pyramid.  The  most  commonly  used  filters  are  the  opening  and 
closing  operations,  though  interesting  results  can  be  obtained  by  combining  the  basic 
operations  in  a  different  manner.  When  a  linear  type  filter  is  used,  the  bandpass  images 
generally  represent  different  portions  of  the  frequency  domain;  but  if  a  morphological  fil¬ 
ter  is  used,  the  bandpass  images  are  representative  of  some  portion  of  the  pattern  spectrum 
in  the  image. 


3.0  Image  Compression 

3.1  General  Compression  Measures 

Two  general  measures  of  image  compressibility  are  entropy  and  variance.  If  pixel  values 
in  an  image  are  assumed  to  be  statistically  independent,  then  the  entropy  of  the  image 
yields  the  minimum  number  of  bits  per  pixel  (bpp)  required  to  exactly  encode  the  image.  It 
is  possible  to  approach  this  value  using  current  compression  techniques.  Our  test  images 
contain  8  bpp  corresponding  to  gray  scale  values  from  0  to  255.  Thus  in  the  worst  case  the 
entropy  of  a  given  image  would  be  8.0  bpp  while  in  the  best  case,  a  si  igle  valued  image, 
the  entropy  would  be  zero. 


255 


H  =  -  x  log ^(i) 

i  =  0 


‘H’  represents  the  image’s  entropy  which  is  calculated  from  the  images  histogram;  ‘f(i)’ 
is  the  normalized  frequency  of  each  gray  scale  value.  In  Burt’s  paper,  he  calculated  the 
entropy  of  ‘USC.girl’  to  be  7.57  bpp;  while  using  my  own  software,  I  determined  the 
entropy  for  the  original  image  to  be  7.59  bpp.  So  my  software  was  reasonably  accurate 
and  supported  their  result. 

Another  representative  measure  of  the  compressibility  of  an  image  is  variance.  For  the 
best  case  of  a  single  valued  image,  the  variance  will  be  zero;  while  for  the  worst  case  for  a 
512x512  image,  the  variance  would  be  16,256.31.  It  must  kept  in  mind  that  these  two 
measures  are  not  directly  related.  For  example,  the  worst  case  for  variance  is  in  fact  an 
image  with  a  very  low  entropy,  1.0  bpp.  The  variance  is  calculated  to  be  the  normalized 
sum  of  the  pixel  differences  from  the  mean  then  squared. 
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The  result  from  Burt’s  paper  for  the  variance  of  USC.girl  is  2736  and  the  result  I  obtained 
for  the  same  image  was  2793.  Again,  the  result  is  not  perfect  but  very  close. 

3.2  Pyramid  Compression 

The  motivation  for  using  the  pyramid  structure  is  supplied  in  Burt  and  Adelson’s  paper  on 
Laplacian  pyramids  Due  to  the  high  correlation  of  neighboring  pixels,  it  is  redundant  to 
store  image  data  as  a  simple  matrix  of  pixel  values.  So  a  format  is  needed  where  the  pixels 
are  decorrelated  if  high  compression  rates  are  to  be  achieved.  In  predictive  coding,  the 
image  is  viewed  as  a  single  stream  of  pixel  values,  raster  format,  and  previous  pixel  values 
are  used  to  estimate  the  next  pixel.  Then  the  error  between  the  predicted  and  actual  value 
is  what  is  stored  for  that  pixel.  Because  prediction  depends  upon  previously  decoded  val¬ 
ues,  this  is  said  to  be  causal  in  nature  [Burt  532].  Non-causal  prediction  based  on  a  neigh¬ 
borhood  should  yield  better  prediction  and  thus  better  compression  results.  Non-causal 
prediction  does  suffer  because  simple  sequential  coding  is  no  longer  possible;  techniques 
such  as  image  transforms  which  encode  blocks  are  generally  used. 

One  point  about  this  type  of  pyramid  structure  that  must  be  considered  if  image  compres¬ 
sion  is  to  be  attempted  is  the  pyramid  is  oversampled.  This  is  one  reason  why  there  has  not 
been  much  effort  put  into  using  the  pyramid  structure  for  data  compression.  The  zeroth 
bandpass  level  contains  the  same  number  of  pixels  as  the  original  image  and  then  you 
must  add  the  rest  of  the  required  levels.  If  we  assume  the  2-D  image  to  be  square  with  sin¬ 
gle  dimension  size  of  n,  then  the  zeroth  level  contains  n2  pixels,  the  first  level  then  has  n2/ 
4,  and  so  on.  If  we  '  i  assume  a  worst  case  where  there  are  the  maximum  number  of  pyr¬ 
amid  levels,  k,  the  total  number  of  pixels  is  a  geometric  series  approaching  4n2/3. 


i  =  0 


3.3  General  Compression  Results 

Part  of  the  original  goal  of  this  project  was  to  compare  the  usefulness  of  nonlinear  and  lin¬ 
ear  image  pyramids  for  image  compression.  Two  general  measures  of  compressibility, 
entropy  and  variance,  were  used  to  compare  the  two  types  of  pyramids.  It  is  already 
known  that  the  linear  type  of  pyramid  can  provide  good  image  compression  despite  the 
fact  that  it  is  over  complete  by  1/3  [Yu  161].  In  general,  the  image  pyramid  decorrelates 
the  pixel  values  leading  to  a  set  of  histograms  as  pictured  in  figure  5. 
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Figure  5. a  is  the  original  USC.girl  and  you  can  see  that  the  image  is  somewhat  random  in 
its  distribution  and  is  thus  very  difficult  to  encode.  Figure  5.b0  shows  the  highest  level 
channel  of  the  Laplacian  pyramid,  Bq,  note  that  the  distribution  is  now  unimodal  with 
entropy  of  4.75  bpp  and  a  variance  of  78.3.  The  next  level  of  theLaplacian  pyramid,  Bl,  is 
shown  in  figure  5.bl  and  has  an  entropy  of  5.07  bpp  with  a  variance  of  153.9.  The  best 
nonlinear  pyramid  performance,  a  closing  by  a  level  3x3  cross  filter,  is  shown  in  Figures 
5.c0,  Bq,  and  5.cl,  Bj.  The  variance  of  the  top  nonlinear  pyramid  level,  BO,  is  90.9  with  a 
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corresponding  entropy  of  4.84  bpp;  the  next  level  of  the  nonlinear  pyramid,  Bl,  has  an 
entropy  of  5.10  bpp  with  a  variance  of  180.8.  Note  that  the  histograms  of  both  the  closing 
and  linear  pyramids  are  very  similar  in  nature  but  have  slightly  different  entropies  and 
variances.  It  is  difficult  to  see  a  difference  at  this  scale  but  the  differences  become  apperent 
when  the  histograms  are  zoomed  in  on.  Due  to  the  nature  of  the  pyramid  structure,  the  top 
level  residual  images  must  be  highly  compressible  if  the  overall  structure  is  to  be  com¬ 
pressible.  The  top  levels  contain  most  of  the  pixels  in  the  image  and  thus  only  the  top  two 
levels  were  examined. 

In  an  attempt  to  obtain  better  results  than  the  Burt’s  Laplacian  pyramid,  we  used  three  dif¬ 
ferent  structuring  elements  of  varying  heights.  The  two  dimensional  shapes  of  the  structur¬ 
ing  elements  starting  at  a  3  x  3  cross  and  increasing  in  size  on  up  to  a  diameter  5  disk. 
Constant  valued  and  Gaussian  shaped  structuring  elements  were  attempted  with  varying 
results.  However,  in  general,  the  smaller  the  structuring  element  was,  with  height  zero,  the 
better  it  performed  with  respect  to  compressibility  measures.  Appendix  A  holds  the  results 
obtained  for  the  different  structuring  elements  and  masks  used.  Several  different  images 
were  also  attempted,  two  cytological  cell  samples,  and  a  slice  from  a  3-D  image  of  a 
guinea  pig  cochlea,  with  the  same  general  results. 

These  results  show  that  the  morphological  pyramid  can  NOT  provide  consistently  better 
image  compression  than  Burt’s  Laplacian  pyramid.  The  results  do  point  out  that  the  Open¬ 
ing  and  Closing  pyramids  always  out  perform  the  Erosion  and  Dilation  types  of  pyramids. 
This  should  be  obvious  as  the  residue  image  represents  the  change  in  the  original  image, 
after  filtering,  down_sizing,  up_sizing,  and  then  interpolating,  and  the  change  is  smaller 
when  Opening  or  Closing  is  done. 


4.0  Single  Value  Decomposition  (SVD) 

Burt  stated  in  his  Laplacian  pyramid  paper  that  the  noncausal  prediction  techniques,  such 
as  pyramids,  were  best  suited  for  image  transforms  such  as  SVD  [Burt  532].  Thus,  the 
next  area  of  investigation  was  to  see  if  the  pyramid  data  structure,  both  linear  and  nonlin¬ 
ear,  could  use  single  value  decomposition  and  recovery  for  data  compression.  This  sec¬ 
tions  goal  was  to  see  if  SVD  could  be  used  for  compressing  the  different  bands  of  the 
pyramid  type  of  data  structure.  To  be  successful,  the  SVD  of  the  pyramid  must  provide 
higher  image  compression  than  the  SVD  of  the  original  image.  A  brief  overview  of  the 
general  SVD  generation  and  reconstruction  theory  followed  by  the  results  obtained  for 
this  section  will  be  given. 

4.1  Transform  Theory 

Single  value  decomposition  is  a  numerical  technique  used  to  diagonalize  matrices  in  an 
attempt  to  achieve  data  compression  [Andrews  425].  Deterministically,  the  SVD  of  an 
image  is  known  to  be  the  optimal  transform  for  energy  compaction  [Jain  179],  as  the 
image  is  decomposed  into  a  set  of  sorted  eigenimages  where  they  are  sorted  by  the  energy 
contained  in  each.  Thus,  by  choosing  only  a  small  number  of  these  eigenimages,  it  is  pos- 


August  22,  1991 


12 


sible  to  attain  a  high  image  compression  ratio  while  minimizing  the  error  due  to  compres¬ 
sion  loss.  This  will  be  later  explained  in  more  detail. 


The  original  image,  shown  by  the  matrix  [G],  can  be  represented  by  a  set  of  eigenvalues, 
or  weights,  and  two  orthonormal  matrices  [U]  and  [V]. 

[G]  =  &/][»]  [if 


‘T’  represents  the  transpose  of  the  ‘V’  matrix.  The  matrix  ‘w’  is  the  eigenvalues  for  the 
decomposed  image  and  is  actually  all  zero  with  the  exception  of  the  main  diagonal  which 
holds  the  sorted  eigenvalues.  There  are  several  methods  available  to  compute  the  ‘U\  ‘w’, 
and  ‘V’  matrices  including  Symmetric  QR,  Hestenes  Method,  Golub-Kahan,....  The 
method  used  in  this  case  was  dictated  by  availability,  ‘Numerical  Recipes  in  C’  provides  a 
decomposition  routine  based  on  the  Golub  method  along  with  a  reconstruction  routine. 

The  reconstruction  of  the  image  from  the  ‘U’,’w’,  and  ‘V’  matrices  is  accomplished  by 
multiplying  each  weight  with  the  multiplication  of  corresponding  vectors  ‘Uj’  and  ‘V,, 
then  adding  up  all  the  new  matrices.  Each  ‘i’  represents  what  is  known  as  an  eigenimage. 
Due  to  the  sorted  nature  of  the  eigenvalues,  the  first  eigenimage  contains  the  most  infor- ' 
mation,  then  the  second  eigenimage,  and  so  on. 

R 

G  = 

i  =  1 


w  j  >  w>2  ^  ^  >  0 


vv 


R+  1 


=  w 


R  +  2 


=  vv 


0 


Depending  on  the  specific  image  being  decomposed,  some  of  the  weights  may  go  to  zero 
and  thus  be  unnecessary  for  full  image  reconstruction.  Some  of  the  weights  will  also  be 
very  small  relative  to  the  highest  energy  weight,  wj.  So  it  is  then  possible  to  use  even 
fewer  than  ‘R’  weights,  ‘K’,  in  the  reconstruction  and  still  recover  the  image  with  a  mini¬ 
mal  amount  of  error.  Again,  because  of  the  energy  packing  property  of  SVD,  this  error  is 
minimized  in  the  following  fashion. 
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G*  =  I  w,xU,xVj 

i  =  1 

R 


Gk  wjUiV,1  w2U2V2t  W3U3V3T  wKUkVKT 


FIGURE  6.  Graphical  representation  of  eigenimages  in  reconstruction 


Figure  6  shows  graphically  how  the  original  image  is  reconstructed  from  the  set  of  eigen¬ 
images.  In  the  figure,  the  level  of  shading  is  representative  of  the  amount  of  information 
present  in  image;  the  darker  the  image,  the  more  information  present. 

4.2  Application  Specifics 

One  of  the  major  problems  with  using  SVD  is  that  the  computation  of  the  matrices 
‘Uy  w\  and  ‘V’  must  all  be  computed  for  each  individual  image  and  this  computation  is 
0(N3)  [Andrews  1  426].  This  is  the  worst  computational  efficiency  of  all  the  well  known 
transform  domains  for  image  representations.  Due  to  this  problem  and  in  the  interest  of 
not  using  all  the  CPU  time,  each  image  is  divided  into  a  set  of  smaller  subblocks  of  dimen¬ 
sion  64  x  64.  Even  with  this  subdivision  of  images,  each  pyramid,  512x512  original 
image,  took  well  over  20  minutes  to  decompose. 

If  the  comparison  is  to  be  accurate,  the  pyramid  decompositions  followed  by  SVD  must 
give  better  results  than  the  SVD  of  the  original  with  the  same  number  of  values  being 
stored.  So,  the  relationship  between  the  number  of  values  for  the  original  and  for  the  pyra¬ 
mid  decomposition  must  be  derived.  Let,  ‘Sy  represent  the  1-D  size  of  the  subblock,  V 
be  the  1-D  size  of  the  square  original  image,  ‘My  be  the  number  of  weights  per  subblock 
used  in  the  recovery  of  the  original  image,  and  ‘Mpj’  represent  the  number  of  weights  per 
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subblock  to  recover  level  ‘i’  of  the  pyramid.  Each  weight  used  corresponds  to  a  single 
eigenimage  and  (2  *  Sj,  +  1)  values  are  required  to  store  each  eigenimage  block. 


n  2 

(£■)  Mo(2Sb+  1)  >  (2 Sb+  1) 

i>b 


(i,  m>+(^)  u, i  + 


..  _  Mp  1  Mp2  Mp3 

Mo  >  MpO  +  -2—  +  -fr-  +  Sr  +  •  • 
4  16  32 


For  simplicity  sake,  if  it  is  assumed  that  Mp  =  MpO  =  Mpl  =  Mp2  = - then  the  summa¬ 

tion  simply  becomes  a  geometric  series.  The  relationship  between  the  number  of  weight 
used  to  recover  the  original  must  be  at  least  4/3  the  number  of  weights  used  to  recover  the 
pyramid.  This  relationship  is  based  on  the  recovered  pyramid  with  the  specified  number  of 
weights  being  of  the  same  or  better  quality  than  the  original  recovered  with  its  correspond¬ 
ing  number  of  weights.  If  this  relation  holds  true,  then  the  pyramid  SVD  outperforms  the 

Mo  >  4 
Mp~  3 


SVD  of  the  original  image. 

The  next  question  to  be  answered  is  what  is  better  quality  defined  to  be.  For  my  work,  I 
chose  to  use  a  combination  of  visual  inspection  and  RMS  error.  RMS  error  is  not  always 
an  ideal  measure  of  image  quality,  but  when  used  in  conjunction  with  visual  inspection,  it 
can  be  a  reasonable  measure  of  image  quality. 

4.3  SVD  Results 

As  previously  stated,  in  order  for  the  pyramid  SVD  to  gain  an  advantage,  it  must  have  the 
same  or  better  image  quality  than  the  SVD  of  the  original  image  using  4/3  the  number  of 
weights  as  the  pyramid  SVD.  For  my  actual  comparison,  see  Figure  7,  the  4/3  factor  was 
not  even  taken  into  account;  if  it  had  been,  the  results  would  be  even  more  obvious. 
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As  image  quality  is  inversely  related  to  the  RMS  error,  Figure7  shows  that  the  relationship 
between  RMS  error  and  the  number  of  weights  used  for  recovery  is  approximately  expo¬ 
nential  in  nature,  (1  -  ae'aM).  However,  the  results  were  not  what  we  were  hoping  to 
attain.  The  SVD  of  the  original  USC.girl  consistently  outperformed  both  the  linear  Lapla- 
cian  pyramid  SVD  and  the  morphological  pyramid  SVD.  There  was  not  a  noticeable  dif¬ 
ference  between  the  linear  and  morphological  S  VDs.  The  morphological  pyramid  used  for 
this  example  was  a  Closing  by  the  5  pixel  diameter  disk;  it  performed  the  best  out  of  the 
nonlinear  filters  attempted. 

Although  the  figure  shows  only  RMS  error,  I  did  visually  compare  the  SVDs  at  various 
points  and  it  appeared  that  the  RMS  measure  was  providing  an  accurate  representation  of 
the  image  quality.  So,  I  can  come  to  no  other  conclusion  than  the  combination  of  SVD  and 
image  pyramid  combination  is  NOT  an  effective  method  for  image  compression. 

Figure  8  shows  the  results  for  the  reconstruction  of  USC.girl,  512x512,  from  (a)  the  raw 
image  data  and  (b)  a  closing  by  5  pixel  diameter  flat  disk  pyramid.  The  raw  image  was 
recovered  using  15  weights  persubblock  while  the  bandpass  pyramid  was  reconstructed 
using  12  weights  per  subblock,  the  4/3  relationship  holds  true.  It  can  easily  be  seen  that  the 
recovery  from  the  raw  data  is  much  clearer  and  of  better  quality. 
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5.0  Pyramids  as  Image  Processing  Platforms 


After  the  lack  of  success  in  the  previous  two  attempts  at  using  morphological  pyramids, 
the  application  of  the  pyramids  as  an  alternative  image  process  platform  was  attempted. 
We  wanted  to  check  the  possibility  that  morphological  pyramid  could  outperform  both  the 
linear  type  of  pyramid  and  the  processing  of  the  raw  image  data.  To  be  successful  at  pro¬ 
viding  an  alternative  image  processing  platform,  some  special  area  or  problem  had  to  be 
better  solved  with  a  morphological  type  of  pyramid. 

First,  the  possible  types  of  pyramid  processing  applications  should  be  reviewed.  These 
operations  are  not  specific  to  morphological  pyramids;  instead,  they  are  applicable  to  any 
type  of  image  pyramid.  Then,  specifically  what  was  accomplished  in  this  area  will  be 
examined. 
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5.1  Potential  Applications 


5.1.1  Independent  Band  Processing 

It  is  possible  to  process  each  band,  or  level,  of  the  pyramid  as  a  separate  image  in  the  same 
way  that  the  raw  image  data  is  processed.  However,  because  of  the  fact  that  each  band  is  a 
portion  of  the  pattern  spectrum  or  frequency  domain,  different  results  will  be  achieved 
after  the  image  is  synthesized.  Thus,  a  more  powerful  image  processing  platform  is 
obtained. 

Some  examples  of  independent  band  processing  include  but  are  in  no  way  limited  to 
multiband  noise  coring  and  multiband  interpolation.  The  noise  coring  operation  can  be 
accomplished  by  taking  each  high  pass  image,  generated  as  the  difference  between  the 
lowpass  image  and  the  previous  lowpass  image,  and  setting  the  pixel  value  to  zero  if  it 
falls  below  a  predetermined  threshold  value.  Then,  when  the  image  is  reconstructed  from 
the  bandpass  and  single  lowpass  image,  the  end  result  will  be  that  much  of  the  high  fre¬ 
quency  noise  will  be  removed  from  the  image.  The  loss  of  thudded  information  will  be 
minimal  in  comparison  to  what  would  be  lost  if  this  operation  were  attempted  at  a  single 
scale  instead  of  a  pyramid  [Lee  16]. 

Multiband  interpolation  can  be  used  to  fill  in  missing  information  in  an  image.  Though 
single  scale  interpolation  operations  exist,  this  method  provides  an  alternative  method  to 
fill  in  the  missing  data.  Figure  9  shows  the  method  of  interpolation  using  a  morphological 
closing  operation.  Note  that  the  missing  data  should  be  set  to  zero  in  both  the  lowpass 
decomposition  pyramid  and  then  in  the  bandpass  pyramid. 


FIGURE  9.  Multiband  Interpolation 
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5.1.2  Cooperative  Band  Processing 

Though  the  independent  band  processing  does  offer  some  unique  possibilities,  a  more 
interesting  application  of  image  pyramids  is  cooperative  band  processing.  Because  of  the 
pyramid  property  that  spatial  locality  is  maintained  between  pyramid  levels,  it  is  possible 
to  perform  operations  between  bands.  This  provides  a  method  of  processing  unavailable 
when  using  raw  image  data. 

Exceeding  information  can  be  removed  from  an  image  through  the  use  of  between  band 
correlation  which  involves  relating  the  data  in  a  given  band  with  other  bands  in  the  image 
pyramid.  Two  types  of  correlation  operations  that  are  available  are  a  weakening  correla¬ 
tion  strategy  and  a  strengthening  correlation  strategy. 

The  weakening  correlation  constrains  the  data  intensity  of  a  given  band  by  taking  the  pixel 
by  pixel  minimum  between  the  given  bands  and  its  adjacent  bands.  It  is  also  possible  to 
preprocess  the  adjacent  bands  with  operations  such  as  morphological  dilation  and  then 
perform  the  weakening  operation.  For  our  application,  the  correlation  was  only  done  with 
one  adjacent  band  Bi+1.  In  this  case,  the  adjacent  band,  Bj+i,  differs  in  size  from  B,  by  a 
factor  of  two  in  each  dimension,  so  it  must  first  be  up-sized  then  interpolated. 


B?  =  Min(BitDilate(P(U(Bi+  j) ) ) ) 


In  this  way,  the  resulting  band  image,  Bj,  is  constrained  to  the  dilated  value  of  its  adjacent 
band,Bi+j.  This  weakening  operation  can  be  applied  iteratively  from  coarse  (fine)  resolu¬ 
tion  down  to  the  fine  (coarse)  resolution  [Lee  1 8 j. 

The  opposite  of  the  weakening  operation  is  enhancement.  Image  features  can  be 
enhanced,  brightened,  by  performing  the  correlation  between  bands;  but,  instead  of  using 
a  minimum  operation  as  in  weakening,  a  maximum  operator  is  used  and  a  type  of  weaken¬ 
ing  is  first  accomplished.  This  operation  results  certain  features  being  enhanced  with  the 
size  and  degree  of  brightening  being  dependant  upon  the  size  and  shape  of  the  structuring 
element  being  used  in  the  dilation  and  the  structuring  element  used  in  the  initial  pyramid 
decomposition. 


B \  =  Max  (B^Min  {Dilate  (£•),  (P(U(Bi+  j))))) 


5.1.3  Selective  Synthesis 

By  leaving  out  certain  bands  in  the  reconstruction  it  is  possible  to  obtain  different  results. 
If  the  goal  is  to  extract  certain  patterns  below  or  above  a  certain  size,  then  by  using  only 
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the  bands  containing  the  desired  patterns  with  the  given  size  in  the  reconstruction  it  is  pos¬ 
sible  to  obtain  an  image  with  the  desired  properties. 


5.2  Specific  Applications  and  Results 

5.2.1  Strip  Artifact 

The  first  area  that  was  investigated  in  hopes  the  morphological  pyramid  would  prove 
superior  was  the  removal  of  a  structured  noise,  a  strip  artifact,  from  an  image  of  a  guinea 
pig  cochlea.  Cochlea,  Figure  lO.a.  This  type  of  artifact  could  possibly  result  from  a  digi¬ 
tizer  with  a  neighborhood  of  photodetectors  that  were  working  incorrectly.  The  strip  arti¬ 
fact  introduced  was  five  pixels  wide  with  random  bidirectional  noise  in  the  strip.Two  types 
of  linear  pyramids  with  different  frequency  properties,  a  pair  of  morphological  pyramids 
and  classical  noise  removal  techniques  on  the  raw  image  data  were  attempted. 

The  morphological  process  used  to  remove  the  strip  involves  the  use  of  two  different  types 
of  nonlinear  pyramids  and  weakening  strategies.  The  first  step  was  to  generate  an  opening 
pyramid  using  a  level  3x3  square  structuring  element.  Then  a  weakening  strategy  using  a 
dilation  by  the  same  structuring  element  of  the  smaller  adjacent  bands  was  used  to  remove 
part  of  the  strip.  The  weakened  bandpass  pyramid  was  then  reconstructed.  The  partially 
improved  image  was  then  again  decomposed.  This  time,  however,  a  closing  pyramid 
based  upon  a  flat  disk,  5  pixels  in  diameter,  was  used.  Then  this  bandpass  pyramid  was 
weakened  using  the  smaller  adjacent  band  dilated  by  the  disk  used  in  the  decomposition. 
Finally,  this  bandpass  pyramid  was  reconstructed  resulting  in  the  best  removal  of  the  strip, 
see  Figure  lO.b. 

The  two  linear  pyramids,  each  having  a  mask  with  different  cutoff  frequencies,  used 
between  band  correlation  to  remove  the  strip.  If  the  recovered  images  from  both  are  exam¬ 
ined,  Figures  lO.c  and  lO.d,  it  can  be  seen  that  both  had  partial  success  in  removing  the 
strip  but  failed  to  completely  remove  it.  Even  in  the  black  area  outside  of  the  cochlea,  the 
strip  is  still  partially  visible. 

The  classical  attempt  used  was  a  repetitive  convolution  with  a  Gaussian  type  of  lowpass 
mask.  As  can  be  seen  in  Figure  lO.e,  this  attempt  also  failed  to  successfully  remove  the 
cochlea  and  predictably  fared  worse  than  the  linear  pyramids. 
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5.2.2  Unstructured  White  Noise 


After  the  morphological  pyramid  proved  successful  in  removing  the  strip  artifact  with  bet¬ 
ter  results  than  the  competing  methods,  the  next  logical  step  was  to  attempt  the  removal  of 
noise  from  the  entire  image.  Unstructured  white  noise,  approximating  speckle  noise,  was 
introduced  to  the  U SC. girl  image  resulting  in  figure  1  l.a.  The  next  step  was  to  attempt  to 
remove  the  noise  using  a  variety  of  techniques. 

An  opening  pyramid  based  on  a  flat  3  x  3  cross  structuring  element  was  generated.  This 
isolated  the  noise  in  the  Bg  image  on  one  side  of  zero.  The  next  step  was  to  perform  a 
weakening  of  this  band  with  a  dilation  by  a  flat  3  x  3  square  of  its  adjacent  band,  Bj,  to 
remove  the  noise.  The  image  was  then  reconstructed  using  the  normal  process  resulting  in 
the  removal  of  the  noise  with  a  minimal  loss  of  image  quality,  see  Figure  1  l.b. 

For  the  two  linear  pyramids,  each  filter  having  a  different  cutoff  frequency,  I  again 
attempted  to  use  between  band  correlation  to  remove  the  noise.  The  weakening  strategy 
had  to  be  iteratively  applied  down  the  image  pyramid  as  the  noise  was  not  isolated  in  one 
band.  This  resulted  in  greater  error  in  the  reconstructed  image  as  the  correlation  operation 
does  introduce  some  error.  Figure  1  l.c  shows  the  result  of  the  first  linear  pyramid  attempt 
and  it  fails  to  completely  remove  the  noise.  The  other  linear  pyramid  attempt  does  succeed 
in  removing  the  noise  but  as  Figure  ll.d  shows,  a  large  amount  of  error  was  introduced. 

The  results  would  be  incomplete  if  an  attempt  was  not  made  to  remove  the  same  noise 
with  classical  techniques  on  the  raw  image  data.  Thus  a  repetitive  covolution  with  a  Gaus¬ 
sian  lowpass  mask  was  attempted.  The  results,  Figure  ll.e,  clearly  showed  that  the  mor¬ 
phological  pyramid  to  be  superior. 

Some  might  argue  that  a  simple  opening  operation  on  the  raw  image  data  would  be  much 
simpler  and  as  effective  as  the  morphological  pyramid.  However,  this  is  not  true.  If  only 
opening  is  done  then  some  information  will  be  lost  as  more  than  just  the  noise  is  removed. 
When  the  pyramid  is  used,  the  noise  is  removed  but  the  other  information  is  added  back 
into  the  image  during  the  reconstruction  process.  Thus  resulting  in  better  performance. 

As  the  (JSC. girl  results  show,  the  morphological  pyramid  proved  superici  to  all  other 
types  of  operations  attempted.  Each  image  has  its  corresponding  RMS  error  which  again 
shows  the  morphological  approach  to  be  the  best  for  this  application 
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6.0  Conclusions 


This  research  was  initiated  to  find  some  application  in  which  morphological  pyramid 
decompositions  would  provide  the  proper  data  representation,  resulting  in  better  results 
than  current  linear  practices.  Three  different  image  processing  applications  of  morpholog¬ 
ical  pyramids  were  investigated:  general  image  compression,  image  pyramid  transforms, 
and  pyramid  processing. 

For  the  general  image  compression  measures,  a  comparison  between  Burt’s  Laplacian  pyr¬ 
amid  and  a  variety  of  morphological  pyramids  showed  that  the  nonlinear  pyramids  could 
not  outperform  the  linear  decomposition.  Though  the  linear  pyramid  consistently  outper¬ 
formed  the  nonlinear  pyramids,  some  of  the  morphological  pyramids  did  provide  close 
results. 

In  the  area  of  using  the  single  value  decompositions  of  image  pyramids  for  compression, 
no  success  was  found.  The  SVDs  of  the  raw  image  data,  a  linear  pyramid,  and  several 
morphological  pyramids  were  examined.  However,  t  he  decomposition  of  the  raw  image 
da  proved  far  superior,  with  respect  to  lossy  compression  error,  to  all  the  pyramids 
attempted. 

The  final  area  of  invest'gation  was  whether  the  nonlinear  pyramids  could  provide  a  power¬ 
ful  image  processing  platform  that  could  compete  with  both  linear  pyramids  and  classical 
raw  image  processing  techniques.  The  results  in  .nation  clearly  showed  the  morpho¬ 
logical  pyramids  could  in  fact  be  used  for  noise  and  artifact  removal  with  much  more  suc¬ 
cess  than  classical  raw  image  processing  and  linear  pyramid  processing. 

Our  original  goal  of  demonstrating  a  specific  application  where  the  nonlinear  image  pyra¬ 
mids  could  provide  an  equal  or  better  method  of  data  representation  was  accomplished. 
The  nonlinear  pyramid  approach  proved  to  be  superior  to  other  linear  practices  in  the  area 
of  noise  and  artifact  removal.  No  other  method  attempted  could  even  come  close  to  the 
results  that  were  obtained  in  this  area. 
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General 


APPENDIX  A 


Compressibility  Results 


LINEAR  DECOMPOSITIONS 
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Morphological  Open/Close  Filters 
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Morphological  Di iate/Erode  Filters 
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